There exists a general need in the development of software/systems to ensure that the finished product is sufficiently tested prior to its release to the public. Such testing is performed to detect programming errors/flaws. Corrections are formulated and then incorporated into subsequent versions of the software. There are various levels of testing thoroughness. The more thorough software is tested prior to release of the software to users, the less likely bugs will be exposed in the subsequent use of the released software.
An application program interface of an operating system comprises a set of methods defining entry points by applications into the functionality (e.g., retrieving/opening files, launching executable programs, printing, etc.) supported/provided by the operating system. Furthermore, the functionality of the operating system comprises groups of related functions referred to herein as components. In the Microsoft WINDOWS operating system, examples of components include: bundled items such as Media Player and Shell (which includes multiple subcomponents such as Active Desktop, Address Bar, Common Dialogs, Control Panel, Desktop Folder, Explorer, File Association, etc.). Application program interface (API) testing ensures that operating system components (e.g., printing, shell, etc.), when called upon by application programs, execute in an intended manner. During API testing, an application program calls a set of methods included in the operating system API. The calls are modified during the course of testing to include various combinations of passed parameter values. The behavior of the called method is observed for each of the application calls.
API testing to ensure that an operating system will operate as intended, when properly performed, is both extensive and exhaustive. Such testing is extremely time consuming with regard to programming the calls to the API methods and with regard to the subsequent analysis of the operating system responses to the test calls. By way of example, API testing comprises submitting various combinations of values via an API to operating system components under test. In many, if not most, cases a large number of input parameters, and the many different potential values assignable to each of those input parameters discourages complete testing of the interaction between applications and the operating system program code via the API. As a result, previous methods for testing operating system components generally settled for less than complete coverage of the potential interactions between all application programs and the components of the operating system accessed through the operating system API.
One general technique for testing operating system components involves writing applications to directly call and test operating system components via their specified API methods. In such case, the application programs are written for the sole purpose of exercising the operating system component(s) under test. Such approach provides a high level of control over testing. By directly programming the calls themselves, a tester is able to tailor a test to include specific inputs to APIs under specific contexts. On the other hand, testing cannot be performed until the test program has been rendered.
Another general technique for testing an operating system API avoids the need to write a particular program to act as the caller to the components under test. This alternative approach involves running application programs and observing their operation (and errors/fault) with regard to the various called components of the operating system via the API. This approach avoids the need to author an application to submit calls to the operating system components. Furthermore, this approach ensures that at least the application used to perform the testing operates properly with regard to the particular operating system API. A potential shortcoming of this approach is the wide variety of applications that execute on an operating system and call upon its functionality through the API.
The above-described approaches to API testing, while offering particular advantages, also exhibit shortcomings including either an inability to exhaustively test an API or alternatively perform such exhaustive testing at a very high price in testing time and resources.